{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 作业二：\n",
    "为MNIST数据集构建一个分类器，并在测试集上达成超过90%的精度 10分\n",
    "提示：KNeighborsClassifier对这个任务非常有效，你只需要找到合适的超参数即可，可对weights和n_neighbors这两个超参数进行网格搜索"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 作业二操作步骤\n",
    "###### 这是一个二元分类器\n",
    "- 获取数据:done\n",
    "- 获取样本X,标签y；将X，y的顺序随机打乱：done\n",
    "- 获取训练集60000；测试集10000：done\n",
    "- 获取一个样本数据：39000：2，之后测试用：done\n",
    "- 数据处理：目前MNIST数据集基本都是被处理好的数据集可以直接使用：done;补充：把y_train;y_test转换成int32类型\n",
    "- 引入K-近邻，做fit:K_近邻默认weights：uniform；n_neighbor=5:可以选择：distance和其他的neighbor数组合\n",
    "- GridSearchCV找到最好的参数{'weights':[],'n_neighbors':[]}\n",
    "- precision，recall评估\n",
    "- 要求是精度大于90%"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import fetch_mldata\n",
    "\n",
    "# 导入数据\n",
    "mnist = fetch_mldata('mnist-original', data_home='./')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X=mnist['data']\n",
    "y=mnist['target']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 拆分数据集\n",
    "X_train,X_test,y_train,y_test=X[:60000,:],X[60000:,:],y[:60000],y[60000:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import sklearn\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 洗牌，重新划分\n",
    "shuffle_index=np.random.permutation(60000)\n",
    "X_train,y_train=X_train[shuffle_index],y[shuffle_index]\n",
    "shuffle_index=np.random.permutation(10000)\n",
    "X_test,y_test=X_test[shuffle_index],y_test[shuffle_index]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 随意确定一个值\n",
    "some_digit=X_train[39000]\n",
    "some_digit_img=some_digit.reshape(28,28)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import matplotlib"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAADKVJREFUeJzt3WGoXOWdx/Hfb7X1ha2oZNRodW82\nyhIVN6lDKLgsLsVq1kDsi4QGCVkom76IsIGAioqRwIosbbNFlsDtemkKjW0gVfNCditScItrySRI\ntRt3I3K3zSbc3BBBkzfF5N8X96Rckztnxplz5sz1//1AmDPnOTPPn9HffWbmOWceR4QA5PNnTRcA\noBmEH0iK8ANJEX4gKcIPJEX4gaQIP5AU4QeSIvxAUpePsrMlS5bExMTEKLsEUpmentapU6fcz7FD\nhd/2A5J+IOkySf8WEc+VHT8xMaFOpzNMlwBKtNvtvo8d+G2/7csk/aukNZJul7TR9u2DPh+A0Rrm\nM/9qSe9HxAcR8QdJP5W0rpqyANRtmPDfJOn38+4fK/Z9iu0ttju2O7Ozs0N0B6BKw4R/oS8VLrk+\nOCImI6IdEe1WqzVEdwCqNEz4j0m6ed79r0g6Plw5AEZlmPAflHSb7WW2vyjpW5IOVFMWgLoNPNUX\nEZ/YfkTSf2huqm8qIn5bWWUAajXUPH9EvCrp1YpqATBCnN4LJEX4gaQIP5AU4QeSIvxAUoQfSIrw\nA0kRfiApwg8kRfiBpAg/kBThB5Ii/EBShB9IivADSRF+ICnCDyRF+IGkCD+QFOEHkiL8QFIjXaIb\nnz/nz58vbd+3b1/Xtp07d5Y+9r333itt7/X4p556qrQ9O0Z+ICnCDyRF+IGkCD+QFOEHkiL8QFKE\nH0hqqHl+29OSPpZ0TtInEdGuoigsHjMzM6XtDz/88MDPbbu0/a233hr4uVHNST5/GxGnKngeACPE\n234gqWHDH5J+YfuQ7S1VFARgNIZ9239PRBy3fZ2k12y/FxFvzD+g+KOwRZJuueWWIbsDUJWhRv6I\nOF7cnpT0kqTVCxwzGRHtiGi3Wq1hugNQoYHDb/tK21++sC3pG5LeraowAPUa5m3/9ZJeKqZjLpe0\nNyL+vZKqANRu4PBHxAeS/qrCWoDP5MEHH2y6hEWNqT4gKcIPJEX4gaQIP5AU4QeSIvxAUvx0N4bS\n65LeYaxYsaK0ff369bX1nQEjP5AU4QeSIvxAUoQfSIrwA0kRfiApwg8kxTw/Sp07d660/dlnn62t\n71WrVpW2L1mypLa+M2DkB5Ii/EBShB9IivADSRF+ICnCDyRF+IGkmOdHqcnJydL2/fv3D/zcV111\nVWn7tm3bBn5u9MbIDyRF+IGkCD+QFOEHkiL8QFKEH0iK8ANJ9Zzntz0laa2kkxFxZ7HvWkk/kzQh\naVrShoj4sL4yUZcjR46Utu/cubO2vteuXVvafvfdd9fWN/ob+X8k6YGL9j0u6fWIuE3S68V9AItI\nz/BHxBuSTl+0e52kPcX2HkkPVVwXgJoN+pn/+og4IUnF7XXVlQRgFGr/ws/2Ftsd253Z2dm6uwPQ\np0HDP2N7qSQVtye7HRgRkxHRjoh2q9UasDsAVRs0/AckbS62N0t6pZpyAIxKz/DbflHSf0n6S9vH\nbH9b0nOS7rN9VNJ9xX0Ai0jPef6I2Nil6esV14IGrFmzprR9ZmamtN32wH2vW7du4MdieJzhByRF\n+IGkCD+QFOEHkiL8QFKEH0iKn+5O7sMP670S++qrr+7aduutt9baN8ox8gNJEX4gKcIPJEX4gaQI\nP5AU4QeSIvxAUszzf87t3r27tP3MmTO19l922e7KlStr7RvlGPmBpAg/kBThB5Ii/EBShB9IivAD\nSRF+ICnm+T8HDh061LXtscceK31sRAzVfsMNN5S2T01NlbajOYz8QFKEH0iK8ANJEX4gKcIPJEX4\ngaQIP5BUz3l+21OS1ko6GRF3FvuekfQPkmaLw56IiFfrKhLl3nzzza5tZ8+eLX3sMEtsS9L9998/\n1OPRnH5G/h9JemCB/bsiYmXxj+ADi0zP8EfEG5JOj6AWACM0zGf+R2z/xvaU7WsqqwjASAwa/t2S\nlktaKemEpO91O9D2Ftsd253Z2dluhwEYsYHCHxEzEXEuIs5L+qGk1SXHTkZEOyLarVZr0DoBVGyg\n8NteOu/uNyW9W005AEaln6m+FyXdK2mJ7WOSdki61/ZKSSFpWtJ3aqwRQA16hj8iNi6w+4UaakEX\nBw8eLG1/+umna+v7xhtvLG1/9NFHa+sb9eIMPyApwg8kRfiBpAg/kBThB5Ii/EBS/HT3IrBr167S\n9o8++qi2vrdu3VravmLFitr6Rr0Y+YGkCD+QFOEHkiL8QFKEH0iK8ANJEX4gKeb5x8Dp0+W/j3r4\n8OERVXKpDRs2NNY36sXIDyRF+IGkCD+QFOEHkiL8QFKEH0iK8ANJMc8/Bvbu3VvafvTo0dr63rFj\nR2n78uXLa+sbzWLkB5Ii/EBShB9IivADSRF+ICnCDyRF+IGkes7z275Z0o8l3SDpvKTJiPiB7Wsl\n/UzShKRpSRsi4sP6Sl28el2P/+STT46okktt3769sb7RrH5G/k8kbY+IFZK+Jmmr7dslPS7p9Yi4\nTdLrxX0Ai0TP8EfEiYg4XGx/LOmIpJskrZO0pzhsj6SH6ioSQPU+02d+2xOSVkn6taTrI+KENPcH\nQtJ1VRcHoD59h9/2lyTtl7QtIvpeHM72Ftsd253Z2dlBagRQg77Cb/sLmgv+TyLi58XuGdtLi/al\nkk4u9NiImIyIdkS0W61WFTUDqEDP8Nu2pBckHYmI789rOiBpc7G9WdIr1ZcHoC79XNJ7j6RNkt6x\n/Xax7wlJz0naZ/vbkn4naX09JS5+Z8+eLW0/c+ZMbX3fcccdpe2XX85V3Vn1/C8fEb+S5C7NX6+2\nHACjwhl+QFKEH0iK8ANJEX4gKcIPJEX4gaSY5B2B559/vrG+X3755dL2K664YkSVYNww8gNJEX4g\nKcIPJEX4gaQIP5AU4QeSIvxAUszzj8Bdd91V2r5///6hnn/Tpk1d25YtWzbUc+Pzi5EfSIrwA0kR\nfiApwg8kRfiBpAg/kBThB5JyRIyss3a7HZ1OZ2T9Adm02211Op1uP7X/KYz8QFKEH0iK8ANJEX4g\nKcIPJEX4gaQIP5BUz/Dbvtn2L20fsf1b2/9Y7H/G9v/bfrv493f1lwugKv38mMcnkrZHxGHbX5Z0\nyPZrRduuiPhufeUBqEvP8EfECUkniu2PbR+RdFPdhQGo12f6zG97QtIqSb8udj1i+ze2p2xf0+Ux\nW2x3bHdmZ2eHKhZAdfoOv+0vSdovaVtEfCRpt6TlklZq7p3B9xZ6XERMRkQ7ItqtVquCkgFUoa/w\n2/6C5oL/k4j4uSRFxExEnIuI85J+KGl1fWUCqFo/3/Zb0guSjkTE9+ftXzrvsG9Kerf68gDUpZ9v\n+++RtEnSO7bfLvY9IWmj7ZWSQtK0pO/UUiGAWvTzbf+vJC10ffCr1ZcDYFQ4ww9IivADSRF+ICnC\nDyRF+IGkCD+QFOEHkiL8QFKEH0iK8ANJEX4gKcIPJEX4gaQIP5DUSJfotj0r6f/m7Voi6dTICvhs\nxrW2ca1LorZBVVnbn0dEX7+XN9LwX9K53YmIdmMFlBjX2sa1LonaBtVUbbztB5Ii/EBSTYd/suH+\ny4xrbeNal0Rtg2qktkY/8wNoTtMjP4CGNBJ+2w/Y/h/b79t+vIkaurE9bfudYuXhTsO1TNk+afvd\nefuutf2a7aPF7YLLpDVU21is3FyysnSjr924rXg98rf9ti+T9L+S7pN0TNJBSRsj4r9HWkgXtqcl\ntSOi8Tlh238j6YykH0fEncW+f5Z0OiKeK/5wXhMRj41Jbc9IOtP0ys3FgjJL568sLekhSX+vBl+7\nkro2qIHXrYmRf7Wk9yPig4j4g6SfSlrXQB1jLyLekHT6ot3rJO0ptvdo7n+eketS21iIiBMRcbjY\n/ljShZWlG33tSupqRBPhv0nS7+fdP6bxWvI7JP3C9iHbW5ouZgHXF8umX1g+/bqG67lYz5WbR+mi\nlaXH5rUbZMXrqjUR/oVW/xmnKYd7IuKrktZI2lq8vUV/+lq5eVQWWFl6LAy64nXVmgj/MUk3z7v/\nFUnHG6hjQRFxvLg9Kekljd/qwzMXFkktbk82XM+fjNPKzQutLK0xeO3GacXrJsJ/UNJttpfZ/qKk\nb0k60EAdl7B9ZfFFjGxfKekbGr/Vhw9I2lxsb5b0SoO1fMq4rNzcbWVpNfzajduK142c5FNMZfyL\npMskTUXEP428iAXY/gvNjfbS3CKme5uszfaLku7V3FVfM5J2SHpZ0j5Jt0j6naT1ETHyL9661Hav\n5t66/mnl5gufsUdc219L+k9J70g6X+x+QnOfrxt77Urq2qgGXjfO8AOS4gw/ICnCDyRF+IGkCD+Q\nFOEHkiL8QFKEH0iK8ANJ/RFJ1pDs5pi2NwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x18c6f6654a8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "plt.imshow(some_digit_img,cmap=matplotlib.cm.binary)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.neighbors import KNeighborsClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "y_train=y_train.astype('int32')\n",
    "y_test=y_test.astype('int32')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 选出为1的\n",
    "y_train_1=(y_train==1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',\n",
       "           metric_params=None, n_jobs=1, n_neighbors=5, p=2,\n",
       "           weights='uniform')"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 距离计算 kn模型\n",
    "kn_clf=KNeighborsClassifier()\n",
    "kn_clf.fit(X_train,y_train_1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 选定值进行测试\n",
    "some_digit=X_train[39000]\n",
    "some_digit=[some_digit]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ True], dtype=bool)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 测试模型\n",
    "kn_clf.predict(some_digit)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import cross_val_score\n",
    "from sklearn.metrics import precision_score\n",
    "from sklearn.metrics import recall_score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 错误\n",
    "- cross_val_score\n",
    "- cross_val_predict\n",
    "\n",
    "- 当区分错误的时候，容易出现错误：Found input variables with inconsistent numbers of samples: [60000, 3]\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 交叉验证\n",
    "y_train_pred=cross_val_score(kn_clf,X_train,y_train_1,cv=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "y_train_score=y_train_pred"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(60000,)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train_1.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.99386666666666668"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 交叉验证 得分\n",
    "y_train_pred[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import cross_val_predict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "y_train_pred=cross_val_predict(kn_clf,X_train,y_train_1,cv=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "precision:95.52%\n",
      "recall:99.23%\n"
     ]
    }
   ],
   "source": [
    "# 精度，召回\n",
    "precision=precision_score(y_train_1,y_train_pred)\n",
    "recall=recall_score(y_train_1,y_train_pred)\n",
    "print('precision:{:.2f}%'.format(precision*100))\n",
    "print('recall:{:.2f}%'.format(recall*100))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import GridSearchCV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 网格搜索，调参https://blog.csdn.net/Kyrie_Irving/article/details/90023615\n",
    "param_grid=[\n",
    "    {'weights':['uniform'],'n_neighbors':[i for i in range(1,11)]},\n",
    "    {'weights':['distance'],'n_neighbors':[i for i in range(1,11)]},\n",
    "]\n",
    "final_kn_clf=GridSearchCV(kn_clf,param_grid,cv=3,\n",
    "                         scoring='accuracy')\n",
    "final_kn_clf.fit(X_train,y_train_1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 验证\n",
    "final_kn_clf.predict(some_digit)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 最佳参数\n",
    "final_kn_clf.best_estimator_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 赋值\n",
    "kn_clf_new=KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',\n",
    "                     metric_params=None, n_jobs=None, n_neighbors=1, p=2,\n",
    "                     weights='uniform')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 交叉验证\n",
    "y_train_pred=cross_val_predict(kn_clf_new,X_train,y_train_1,cv=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 精度，召回 \n",
    "precision=precision_score(y_train_1,y_train_pred)\n",
    "recall=recall_score(y_train_1,y_train_pred)\n",
    "print('precision:{:.2f}%'.format(precision*100))\n",
    "print('recall:{:.2f}%'.format(recall*100))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "y_test=y_test.astype('int32')\n",
    "y_test_1=(y_test==1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 测试集  进行验证\n",
    "y_test_pred=cross_val_predict(kn_clf_new,X_test,y_test_1,cv=2)\n",
    "precision=precision_score(y_test_1,y_test_pred)\n",
    "recall=recall_score(y_test_1,y_test_pred)\n",
    "print('precision:{:.2f}%'.format(precision*100))\n",
    "print('recall:{:.2f}%'.format(recall*100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Conclusion\n",
    "- KNeighborsClassfier\n",
    "    - precision：0.9878112484764061\n",
    "    - recall：0.9521651560926485\n",
    "    \n",
    "- GridSearchCV:best_estimator:\n",
    "    - precision:98.23%\n",
    "    - recall:96.04%\n",
    "- test_score\n",
    "    - precision:97.59%\n",
    "    - recall:93.99%"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
